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FOREWORD 


Document  Version  Control 


This  document  supercedes  the  June  2001  (Version  1.0)  publication  of 
SIAP  SE  TF  Technical  Report  2001-001:  Single  Integrated  Air  Picture  Attributes. 


This  document  is  one  in  a  series  of  four  Technical  Reports  on  the  subject 
of  SIAP  Metrics,  all  of  which  are  scheduled  for  regular,  approximately  annual 
updates.  For  most  effective  use,  the  most  recent  version  of  each  document 
should  be  consulted.  As  of  this  writing  (August  2003),  the  latest  version  and  next 
scheduled  update  for  the  other  three  related  technical  reports  are  as  follows: 


SIAP  SE  TF  Technical  Report  Current  Version  (as  of  Aug  03)  Next  Update 


2001-002  SIAP  MOEs  and  MOPs  October  2001 

2001- 003  SIAP  Metrics  Implementation  October  2001 

2002- 007  Ballistic  Missile  SIAP  Metrics  November  2002 


August  2003 
September  2003 
November  2003 
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EXECUTIVE  SUMMARY 


PROBLEM 

The  SIAP  Implementation  Plan  states  that  the  overarching  objective  of  the  SIAP 
SE  TF  is  "to  identify  incremental  improvements  in  the  . . .  SIAP  capability  that  will 
provide  commensurate  incremental  improvements  in  warfighter  capabilities’* 
(SIAP  Implementation  Plan,  2001).  Quantification  of  the  SIAP  capability  in 
evaluative,  predictive,  and  prescriptive  terms  is  a  necessary  step  towards 
assessment  of  such  improvements.  Multiple  measures  currently  exist  to  evaluate 
the  SIAP,  but  there  is  much  overlap  and  some  potential  for  conflict  and  confusion 
between  similar  measures.  Furthermore,  many  of  these  existing  measures  have 
not  been  precisely  quantified,  or  have  been  quantified  with  no  apparent  predictive 
or  prescriptive  goats  in  mind.  Therefore,  there  is  a  clear  need  to  consolidate  and 
more  rigorously  define  a  core  set  of  attributes,  which  provide  the  joint  community 
with  a  common  point  of  departure  for  quantifying  and  assessing  a  SIAP.  These 
attributes  must  be  traceable  to  Theater  Air  and  Missile  Defense  (TAMD)  and 
Combat  Identification  (CID)  Capstone  Requirements  Document  (CRD)  Key 
Performance  Parameters  (KPPs),  augmenting  the  latter  where  either  quantitative 
precision  or  a  sharper  focus  on  warfighting  benefit  is  necessary.  The  SIAP  SE 
TF  Technical  Reports  2001-001,  2001-002,  2001-003,  and  2002-007  were 
developed  by  the  SIAP  SE  TF  to  correct  for  the  deficiencies  cited  above  by 
providing  a  standard  set  of  definitions  and  algorithms  for  quantitative  evaluation 
of  air  picture  quality  within  the  TAMD  community.  The  measures  apply  to 
models,  simulations,  hardware  and  software  in  the  loop  experiments,  operator  in 
the  loop  exercises,  and  field  exercises  and  evaluations.  This  revision  is  based 
on  experience  gained  within  the  community  from  application  of  those  technical 
reports  across  the  spectrum  of  those  venues. 


OBJECTIVES 

The  goal  is  to  establish  a  standard  methodology  for  evaluation  of  the  SIAP. 
based  on  a  minimal  but  sufficient  set  of  SIAP  attributes.  The  attributes  are  to  be 
accessible  to  measurement,  and  will  be  used  to  relate  proposed  engineering 
improvements  to  warfighting  capability. 


APPROACH  AND  SCOPE 

Develop  verbal  definitions  of  SIAP  attributes  and  mathematical  definitions  of 
supporting  SIAP  attribute  measures  that  are  derived  from  CRD  requirements  and 
associated  KPPs.  Identify  and  define  any  additional  attributes  required  to 
comprehensively  characterize  the  SIAP.  The  attributes  and  metrics  in  this  report 
primarily  address  the  SIAP  KPPs  for  air  breathing  objects  as  defined  in  the 
TAMD  CRD.  SIAP  SE  TF  Technical  Report  2002-007  provides  attributes  and 
metrics  for  ballistic  missile  objects. 
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FINDINGS 

The  derivations  and  definitions  of  the  enclosed  SIAP  attribute  measures 
correspond  to  the  TAMD  and  CID  CRD  KPPs.  It  is  reasonable  to  assume  that  as 
SIAP  system  engineering  expertise  deepens,  these  definitions  will  also  evolve  to 
reflect  that  broadened  expertise. 


CONCLUSIONS 

These  attributes  will  form  the  basis  for  measuring  SIAP-related  qualities.  The 
defined  SIAP  attributes  will  enable  meaningful  comparison  of  proposed 
engineering  improvements  and  provide  a  universal  reference  frame  for  greater 
SIAP  community  awareness  and  communication. 


RECOMMENDATIONS 

The  Draft  TAMD  CRD  that  is  in  the  JROC  approval  cycle  requires  that  the 
requirements  set  forth  in  the  KPPs  in  that  document  be  based  on  measurements 
set  forth  in  the  SIAP  SE  TF  Technical  Reports  2001-001, 2001-002,  2001-003, 
and  2002-007.  The  definitions  of  the  air  vehicle  SIAP  attributes  as  refined  in  this 
report  should  fully  support  these  CRD-mandated  measurements  for  the  air 
vehicle  component  of  the  SIAP.  Use  these  definitions  to  determine  the  impact  of 
SIAP  engineering  improvements  at  the  system/unit  level  on  measurable 
warfighting  capability  at  the  Joint  Forces  Command  level.  Update  this  report  on 
an  annual  basis,  to  reflect  the  further  evolution  of  the  SIAP  SE's  assessment 
methodology  and  lessons  learned  from  actual  use  of  the  SIAP  attributes. 
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1.  INTRODUCTION 


This  report  establishes  the  standard  methodology  for  evaluation  of  the 
quality  of  the  air  vehicle  portion  of  the  Single  Integrated  Air  Picture  (SIAP). 
Evaluations  will  be  based  on  a  set  of  SIAP  attributes  for  which  both  qualitative 
definitions  and  mathematical  formulations  of  a  supporting  set  of  measures  are 
provided.  These  attributes  and  their  associated  quantitative  measures  are  also 
needed  to  predict  and  prescribe  SIAP  improvements  that  translate  into  improved 
warfighting  capability. 

SIAP  attributes  measure  the  adequacy  and  fidelity  of  information  which  is 
used  to  form  a  shared  understanding  of  the  tactical  situation  (i.e.,  Situational 
Awareness  (SA)),  and  to  support  battle  management  (BM)  and  engagement 
operations.  As  defined  and  quantified  here,  the  attributes  are  meant  to  provide 
an  approach  to  the  challenge  of  linking  system/unit  level  engineering 
performance  to  quantifiable  warfighting  capabilities. 

The  SIAP  SE  TF  will  use  a  collection  of  tools  to  define  and  develop  a  SIAP 
system  engineering  approach  and  methodology.  Tools  include  models, 
simulations,  hardware  in  the  loop  (HWIL)  experiments,  operator  in  the  loop 
(OITL)  experiments,  and  live  exercises.  These  tools  can  be  used  to  measure  a 
wide  variety  of  characteristics  throughout  a  prescribed  metrics  hierarchy.  Some 
tools  provide  a  high  fidelity  assessment  of  system  or  subsystem  performance  as 
a  function  of  engineering  configuration.  These  tools  expedite  the  determination 
of  the  root  causes  of  SIAP  deficiencies.  A  suite  of  potential  tools  will  also  be 
required  to  characterize  and  assess  the  warfighting  benefit  derived  from  a  robust 
SIAP.  A  few  tools,  such  as  live  exercises  or  evaluations  (e.g.,  Roving  Sands, 

Joint  Combat  Identification  Evaluation  Team  exercises,  etc.),  span  the  entire 
metric  range,  and  they  can  relate  hardware  configurations  to  specific  warfighting 
capabilities.  Typically  there  will  be  fewer  opportunities  to  employ  live  exercises 
due  to  schedule  and  cost  considerations.  HWIL  facilities  offer  the  potential  for 
repeatable  measurements  on  existing  systems  at  a  lower  cost.  Models  and 
simulations  are  especially  useful  when  testing  proposals  for  new  systems  or 
system  changes,  and  can  be  activated  at  even  lower  costs.  A  particularly 
essential  component  of  the  SIAP  SB's  assessment  strategy  for  the  next  few 
years  will  be  the  continuing  development  of  the  Joint  Distributed  Engineering 
Plant  (JDEP)  Technical  Framework  (SIAP  lAP,  2003).  The  JDEP  Technical 
Framework  can  be  broadly  defined  as  a  capability  to  link  distributed  components 
in  user-tailored  federations  which  may  span  across  HWIL,  software-in-the-loop, 
and  simulation  venues  (Dahmann,  2002).  Simulations  will  in  most  cases  need  to 
be  calibrated  to  live  or  HWIL  results  to  provide  confidence,  and  a  mature  JDEP 
Technical  Framework  may  provide  the  means  to  ultimately  migrate  from  HWIL  to 
well-calibrated  and  cost-efficient  simulation  environments.  No  single  tool  is 
adequate  to  meet  all  the  needs  of  the  SIAP  system  engineering  process. 

Therefore  the  tools  will  be  used  in  conjunction  with  the  SIAP  attributes,  the 
attributes  being  the  means  by  which  the  results  of  one  tool  are  transformed  into 
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another.  For  example,  one  model  may  be  used  to  calculate  the  improvement  in 
one  or  more  SIAP  attributes  due  to  change  in  the  correlation  algorithm  used  by  a 
particular  system,  and  another  model  may  be  used  to  calculate  the  warfighting 
benefit  from  that  level  of  SIAP  improvement. 

The  Joint  Requirements  Oversight  Council  (JROC)  has  validated  several 
Capstone  Requirement  Documents  (CRDs)  with  particular  relevance  to  the  SIAP. 
Of  these,  the  most  pertinent  are  the  Theater  Air  and  Missile  Defense  (TAMD) 
CRD  (2001),  the  Combat  Identification  (CID)  CRD  (2001),  the  Information 
Dissemination  Management  (IDM)  CRD  (2001),  and  the  Global  Information  Grid 
(GIG)  CRD  (2001).  These  documents  establish  high-level  performance 
requirements  for  Joint  Forces  in  the  form  of  Key  Performance  Parameters 
(KPPs)  and  related  requirements.  For  each  KPP  relevant  to  the  air  vehicle 
component  of  the  SIAP,  for  which  either  the  TAMD  or  CID  CRD  sets  a 
quantitative  requirement,  there  is  a  corresponding  SIAP  attribute.  The  TAMD 
CRD  also  includes  "Timeliness"  as  part  of  a  SIAP  KPP,  with  associated 
quantitative  requirements  to  be  determined.  Because  SIAP  completeness, 
clarity,  and  accuracy  (three  of  the  SIAP  attributes  defined  in  this  report) 
presuppose  timely  track  data,  a  certain  quality  of  timeliness  is  implied  whenever 
appropriate  goals  are  met  with  regard  to  the  other  attributes.  For  this  reason, 
timeliness  is  not  regarded  here  as  a  separate  attribute.  The  issue  is  discussed  in 
greater  detail  in  Section  2. 3. 1.5  below,  and  will  be  readdressed  whenever  a 
revision  to  one  of  the  aforementioned  CRDs  establishes  a  quantitative  timeliness 
requirement. 

The  TAMD  CRD  prescribes  SIAP  support  to  three  TAMD  functional  areas: 
SA,  BM,  and  advanced  engagement  operations.  The  characteristics  of  the 
information  required  (as  expressed  in  terms  of  attribute  values)  will  vary  with  the 
functional  areas  supported,  as  may  the  information  providers  and  users.  The 
definitions  and  descriptions  in  this  report  are  oriented  toward  the  general  SA  and, 
to  a  lesser  extent,  the  BM  function.  Assessment  of  advanced  engagement 
operations  may  require  some  modification  of  the  definitions  described  in  this 
report.  A  more  in-depth  discussion  of  the  three  TAMD  areas  is  included  in  the 
SIAP  Integrated  Assessment  Plan  (lAP)  (2003). 

The  basic  qualitative  definitions  of  the  SIAP  attributes  presented  in 
Section  2.2  of  this  report  are  intended  to  be  generally  applicable  to  all  SIAP 
tracks  -  broadly  understood  as  conveying  information  on  aerospace  objects, 
ballistic  and  exoatmospheric  objects  included.  However,  the  quantitative 
measures  and  supporting  definitions  addressed  in  this  report  are  only  intended  to 
apply  to  tracks  on  unitary  aerospace  objects  that  do  not  split  into  multiple  objects 
(as  many  types  of  ballistic  missiles  do).  This  limited  scope  covers  the  air  vehicle 
component  of  the  SIAP  (that  is,  tracks  on  tactically  significant  airborne  objects  of 
interest).  While  many  of  the  attributes  defined  in  this  report  are  expected  to  be 
applicable  to  ballistic  missile  defense  (BMD)  as  well,  issues  specific  to  BMD 
require  more  detailed  considerations,  particularly  with  regard  to  reporting  criteria, 
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non-unitary  missiles,  and  ballistic  missile  debris.  Such  considerations  have  been 
relegated  to  a  separate  report  -  SIAP  SE  TF  Technical  Report  2002-007:  Ballistic 
Missile  SIAP  Metrics.  That  report  presents  a  complete  set  of  quantitative  metrics 
applicable  to  the  ballistic  missile  component  of  the  SIAP. 

The  SIAP  SE  TF  will  employ  a  minimal  set  of  attributes  to  characterize  the 
shared  information.  There  are  eight  SIAP  attributes  described  in  this  report.  The 
relationship  of  these  eight  Attributes  to  the  TAMD  and  CID  CRD  KPPs  is 
illustrated  in  Figure  1  below,  and  is  explained  more  thoroughly  in  Appendix  A. 
Italicized  words  in  Figure  1  refer  to  KPPs  and  attributes  that  do  not  have  direct 
correspondence. 


TAMD  CRD  KPPs 

•  oomplftenf'bs 

*  Ai’ibiiuly  (CMrity) 

Figure  1 .  Relationship  between  SIAP  attributes  defined  in  this  report  and 
associated  TAMD  and  CID  CRD  Key  Performance  Parameters 

It  should  be  noted  that  this  table  of  correspondences  applies  to  the  SIAP 
attributes  in  relation  to  the  currently  valid  (2001  publication)  TAMD  and  CID 
CRDs.  The  correspondence  is  not  quite  complete,  and  even  though  this  report 
and  a  CRD  KPP  may  employ  an  attribute  in  the  same  qualitative  sense,  there 
may  be  some  differences  in  the  way  the  metrics  are  quantified.  The  SIAP  SE  TF 
is  presently  working  with  the  services  and  other  organizations  responsible  for 
proposing  CRD  revisions,  with  the  goal  of  bringing  the  CRDs  and  this  technical 
report  into  exact  quantitative  agreement  on  SIAP  metrics  with  the  next  revision 
cycle  (a  revised  TAMD  CRD  is  in  draft  as  of  this  writing). 

The  TAMD  and  CID  CRDs  set  quantitative  requirements  in  terms  of 
threshold  and  objective  values,  which  are  expected  to  be  refined  as  the  analysis 
supporting  them  evolves.  It  is  expected  that  the  objective  and  threshold  values 
of  each  attribute  measure  should  ultimately  depend  on  the  operational  context.  A 
common  operational  context  provides  such  vital  considerations  as  environmental 
factors,  the  Red  threat  details.  Blue  force  laydown,  C2  hierarchy,  rules  of 
engagement,  and  other  criteria.  A  collaboratively  defined.  Defense  Planning 
Guidance  (DPG)-based  "Common  Reference  Scenario"  (CRS)  provides 
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stressing  scenarios  and  mission  level  vignettes  for  SIAP-related  integrated 
system  evaluation.  The  CRS  forms  the  basis  for  developing  the  SIAP 
component  of  an  integrated  TAMD.  SIAP  attributes  will  be  measured  in  context. 
That  is,  information  has  relevance  to  a  user,  with  respect  to  a  mission,  in  an 
operational  context,  with  a  defined  architecture. 

The  objectives  of  this  report,  then,  are  to  define  and  derive  jointly  agreed 
to  SIAP  attributes  that  flow  from  TAMD  CRD  and  CID  CRD  KPPs,  and  to  provide 
a  vehicle  to  institutionalize  the  SIAP  attributes. 

1.1  The  SIAP  Metrics  Hierarchy:  General  Definitions 

The  eight  SIAP  attributes  defined  in  this  report  are  quantifiable,  testable, 
and  measurable.  They  measure  the  quality  of  the  shared  information  available  to 
Joint  and  Combined  Forces  and  are  fundamental  to  assessing  warfighting 
effectiveness.  They  do  not,  however,  exhaust  the  full  range  of  quantitative 
metrics  which  the  SIAP  SE  will  need  to  assess  the  warfighting  effectiveness  of 
specific  proposed  SIAP  improvements.  Figure  2  gives  a  sense  of  the  role  of  the 
SIAP  attributes  as  a  "middle  layer"  in  a  SIAP  metrics  hierarchy,  linking 
system/unit-level  Measures  of  Performance  (MOPs)  to  warfighting  Measures  of 
Effectiveness  (MOEs).  Examples  of  measures  at  each  level  are  provided,  and 
definitions  of  the  levels  follow. 

MOEs 

Fratricide 

-  Leakers 

-  Weapon  Expenditures 

-  Others  (TBD) 

■  Attributes 

Completeness  (2  Attributes) 

Clarity  (2  Attributes) 

Continuity 
Accuracy 
Commonality 

MOPs 

Correlation  Performance 

-  Data  Exchange  Latency 
Navigation  Errors 

“  Sensor  Uncertainty 

-  Messages  Received 

-  Message  Update  Rate 

-  Others  (TBD) 


Warfigh^r 
Benems 


MOE 


SIAP  Attributes 

Systems 

[aegis 


System,  Su 
MOPs 


\  TACtL  1  \ 
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n 
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Figure  2.  SIAP  attributes  and  their  rote  in  the  metrics  hierarchy 
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For  this  report,  MOEs,  SIAP  attributes,  and  MOPs  will  be  defined  as 
follows: 

MOE  -  measure  of  operational  success  that  must  be  closely  related 
to  the  objective  of  the  mission  or  operation  being  evaluated.  (DSMC 
Glossary,  2001) 

A  meaningful  MOE  must  be  quantifiable  and  a  measure  to  what  degree  a  mission 
objective  is  achieved. 

SIAP  Attribute  -  a  quantifiable  property  of  a  SIAP  that  is  derived 
from  TAMD  and  CID  CRD  requirements  and  associated  KPPs. 

MOP  -  measure  of  a  system's  technical  performance,  for  example, 
expressed  as  speed,  payload,  range,  time  on  station,  frequency,  or 
other  distinctly  quantifiat3le  performance  feature.  (DSMC  Glossary, 

2001) 

MOEs,  attributes,  and  MOPs  together  make  up  the  SIAP  metrics 
hierarchy.  MOEs  describe,  in  warfighting  terms,  the  benefit  of  achieving  a  SIAP 
(note,  however,  that  there  may  be  motivations  other  than  warfighting  -  for 
example,  commercial  air  applications).  Fratricide  rate  and  weapons  efficiency 
are  examples  of  MOEs  that  are  expected  to  improve  with  improved  air  picture 
quality.  The  SIAP  attributes  provide  a  standard  means  of  characterizing  the 
SIAP.  As  already  discussed,  the  SIAP  attributes  derive  from  the  TAMD  and  CID 
CRDs.  The  MOPs  quantify  aspects  of  system  and  subsystem  performance  that 
may  be  used  in  the  analysis  of  SIAP  shortfalls  or  the  prescription  of  SIAP 
improvements.  The  ultimate  goal  of  prescribing  engineering  performance 
improvements  which  result  in  significant  warfighting  benefit,  translates 
analytically  into  identifying  feasible  changes  in  system-level  MOPs  that  correlate, 
through  a  testing  and  modeling  process,  with  meaningful  improvements  in  MOEs. 
Because  schedule  and  cost  considerations  will  limit  opportunities  to  use  test 
venues  that  allow  measurements  across  the  entire  hierarchy,  most  tools  will  use 
some  degree  of  modeling  and  simulation  and  will  make  the  linkage  between 
MOPs  and  MOEs  through  the  intermediate  level  of  SIAP  attributes. 

Further  discussion,  and  examples,  of  MOEs  and  MOPs  can  be  found  in 
SIAP  SE  TF  Technical  Report  2001-002.  Definitions  of  the  MOPs  that  will 
actually  be  used  in  connection  with  various  SIAP  SE-sponsored  test  events  are 
being  incorporated  into  updates  of  the  SIAP  Standard  Data  Management  and 
Analysis  Plan  (2002,  March  draft)  as  the  events  are  planned. 
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2.  SIAP  Attributes 


2.1  Definition  of  Terms  and  Assumptions 

The  following  terms  are  either  used  in  the  definitions  of  the  SIAP  attributes 
or  are  generally  relevant  to  the  discussion  of  SIAP  assessments. 

Common  Reference  Scenario  (CRS^  -  a  DPG-based  operational 
context  for  SIAP  assessments,  providing  a  consistent  baseline  for 
evaluating  current  performance  and  proposed  improvements. 

Area  of  Influence  fAOl)  -  “A  geographical  area  wherein  a  commander  is 
directly  capable  of  influencing  operations  by  maneuver  or  fire  support 
systems  normally  under  the  commander’s  command  or  control"  (JP  1-02, 
2001).  The  definition  of  an  AOI  will  be  further  refined  in  the  CRS  to  bound 
the  area  within  which  SIAP  attributes  will  be  evaluated.  The  AOIs  in  the 
CRS  may  be  broken  down  into  limited  participant/mission  specific  AOIs  for 
the  assessment  of  attributes  for  a  given  participant  or  mission. 

Reporting  Criteria  -  specify  the  objects  that  are  to  be  included  in 
the  metric  calculations  and  are  derived  from  the  TAMD  CRD.  A 
complete  description  will  be  prescribed  in  future  updates  to  SIAP 
SE  TF  Technical  Report  2001-003. 

Object  -  for  the  air  track  portion  of  the  SIAP,  any  cruise  missile, 
fixed  wing  aircraft,  rotary  wing  aircraft,  air-to-surface  missile,  large 
caliber  rocket,  unmanned  air  vehicle,  or  other  airborne  object 
meeting  air  vehicle  reporting  criteria,  in  the  area  of  influence.  (The 
object  set  for  the  ballistic  missile  portion  of  the  SIAP  is  addressed  in 
Technical  Report  2002-07.) 

Track  -  "(1 )  the  graphic  and/or  alphanumeric  representation  of  an 
object,  point,  or  bearing  whose  position  and/or  characteristics  are 
collated  from  sensors  and/or  other  data  sources;  (2)  a  collated  set 
of  data. ..associated  with  a  track  number  for  the  purpose  of 
representing  the  position  and/or  characteristics  of  a  specific  object, 
point,  or  bearing"  (MIL-STD6011B,  1999).  The  attributes  and 
metrics  defined  in  this  report  are  based  on  the  definition  of  a  track 
in  the  second  sense. 

Track  Number  -  A  number,  applied  to  a  located  object,  used  to 
associate  information  and  directives. 

For  purposes  of  metric  calculations,  the  track  set  consists  of  actionable  tracks 
where  a  track  is  considered  actionable  once  it  has  an  associated  track  number 
and  is  displayabie  to  an  operator.  Metric  calculations  for  two  of  the  SIAP 
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attributes  -  continuity  and  commonality  -  make  explicit  use  of  track  numbers. 
These  particular  metrics  must  be  understood  in  a  network-specific  sense,  since 
the  sharing  of  track  numbers  between  participants  may  occur  over  several 
networks,  each  of  which  may  have  its  own  numbering  scheme.  The  network 
chosen  for  assessment  determines  the  type  of  track  number  to  be  used.  For 
example,  if  continuity  and  commonality  are  to  be  based  on  Link-16,  the 
appropriate  number  is  the  Link  Track  Number  (LTN). 

Pending  Track  -  "A  track  which  has  not  been  subjected  to  the 
identification  process"  (MIL-STD-6011B,  1999).  Pending  tracks  are 
regarded  as  actionable,  and  therefore  count  towards  the  evaluation 
of  all  SIAP  attributes. 

Assigned  Track  -  in  after-the-fact  analysis  of  track  data  from  an 
exercise  or  simulation,  a  track  which  meets  a  consistent  condition 
for  association  with  an  object  (addressed  in  the  SIAP  Metrics 
Implementation  Technical  Report,  2001-003).  The  assignment  of 
tracks  to  objects  may  be  time  dependent. 

A  purpose  of  the  SIAP  is  to  achieve  a  correct  and  consistent 
understanding  between  participants  within  an  AOI.  A  participant’s  understanding 
is  based  on  information  that  can  be  displayed.  For  this  reason  SIAP  attributes  are 
based  on  data,  held  in  the  central  track  stores  of  the  host  system,  that  are 
displayable  to  the  participant  (see  Figure  3,  below).  This  data  may  come  from 
local  sensor  measurements  as  well  as  remote  sources  such  as  Link-16,  Link-1 1 , 
or  any  other  tactical  data  network. 


Sensor 


Goal: 


Consistent  Understanding  Between  Participants 
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Display 
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Buffers, 
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Attributes  are  measured  here 

Figure  3.  Measurement  of  SIAP  attributes 
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This  report  focuses  on  the  definitions  of  the  SIAP  attributes;  issues 
involving  their  implementation  will  not  be  considered  in  detail,  but  will  now  be 
briefly  summarized.  There  are  a  number  of  issues  related  to  the  determination  of 
which  tracks  will  be  included  in  SIAP  assessments.  Tracks  generated  locally  as 
well  as  those  held  through  remote  sources  should  be  considered  candidate 
tracks  for  scoring  purposes,  but  the  establishment  of  a  definitive  track  base  for 
current  SIAP  assessment  requires  a  more  thorough  treatment  than  can  be  given 
here.  There  is  a  recognized  need  to  discount  certain  “mutual”  tracks  so  that 
repeated  counting  of  essentially  duplicate  information  does  not  bias  the  SIAP 
attributes.  Other  issues  arise  relating  to  the  assignment  of  tracks  to  objects.  An 
assignment  procedure  must  be  defined  to  ensure  consistent  scoring,  and  it  may 
need  to  be  automated  for  modeling  and  simulation  purposes  and  for  some 
experimental  tests.  However,  it  is  beyond  the  scope  of  this  report  to  recommend 
a  specific  assignment  algorithm,  prescribe  a  general  procedure,  or  set  procedural 
limits.  There  are  other  implementation  issues  specific  to  certain  live  exercises 
and  established  scoring  methods.  SIAP  SE  TF  Technical  Report  2001-003 
discusses  all  of  the  above  issues  in  detail. 

2.2  Word  Definitions  of  SIAP  Attributes 

The  definitions  of  the' attributes  and  associated  measures  are  described 
below: 


Completeness:  The  measure  of  the  portion  of  true  air  objects  that 
are  included  in  the  SIAP.  The  air  picture  is  complete  when  all 
objects  are  detected,  tracked  and  reported. 

Clarity:  The  measure  of  the  portion  of  the  SIAP  that  contains 
ambiguous  tracks  and/or  spurious  tracks.  The  air  picture  is  dear 
when  it  does  not  include  ambiguous  or  spurious  tracks. 

Continuity:  The  measure  of  how  accurately  the  SIAP  maintains 
track  numbers  over  time.  The  air  picture  is  continuous  when  the 
track  number  assigned  to  an  object  does  not  change. 

Kinematic  Accuracy:  The  measure  of  how  accurately  the  TAMD 
Family  of  Systems  (FoS)  reports  track  position  and  velocity.  The 
air  picture  is  kinematically  accurate  when  the  position  and  velocity 
of  each  assigned  track  agree  with  the  position  and  velocity  of  the 
associated  object. 

ID  Completeness:  The  measure  of  the  portion  of  tracked  objects 
that  are  in  an  identified  state.  The  ID  is  complete  when  all  tracked 
objects  are  in  an  identified  state. 
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ID  Correctness:  The  measure  of  the  portion  of  tracked  objects  that 
are  in  the  correct  ID  state.  The  ID  is  correct  when  all  tracked 
objects  are  in  the  correct  ID  state. 

ID  Clarity:  The  measure  of  the  portion  of  tracked  objects  that  are 
unambiguously  identified.  The  ID  is  clear  if  no  tracked  object  is  in 
the  ambiguous  ID  state. 

Commonality:  The  measure  of  consistency  of  the  air  picture  held  by 
TAMD  FoS  participants.  The  air  picture  is  common  when  the 
assigned  tracks  held  by  each  participant  have  the  same  track 
number,  position,  and  ID. 

2.3  Complete  Definitions  of  SlAP  Attribute  Measures 

Complete  definitions  of  the  SlAP  attribute  measures  are  given  in  the  following 
sections.  Each  attribute  is  given  a  precise  mathematical  formulation  from  which, 
in  most  cases,  the  following  four  levels  of  metrics  can  be  derived: 

Instantaneous  system  metric  -  the  measure  of  the  attribute  at  an 
instant  in  time  from  the  perspective  of  a  single  participant.  (Omitted 
in  the  case  of  Continuity,  for  which  there  is  no  meaningful 
instantaneous  sense.  Also  omitted  for  Commonality  which  is  an 
inherently  multi-participant  attribute.) 

Time-averaged  system  metric  -  a  time  average,  typically  weighted 
according  to  an  object  or  track  count,  of  the  instantaneous  system 
metric.  (Omitted  in  the  case  of  Commonality,  where  it  would 
coincide  with  the  time-averaged  IADS  metric.) 

Instantaneous  IADS  metric  -  an  average  over  participants,  typically 
weighted  according  to  an  object  or  track  count,  of  the  instantaneous 
system  metric  at  a  particular  instant  in  time. 

Time  -  averaged  IADS  (“roll-up’h  metric  -  a  weighted  average  of  an 
attribute  over  participants  and  time.  For  Continuity  (which  is  not  supported 
by  an  instantaneous  system  metric),  the  weighted  average  is  over 
participants.  For  Commonality,  the  IADS  metric  is  the  time  average  of  the 
consistency  of  the  air  picture  across  participants.  The  IADS  metrics 
provide  a  high  level  assessment  of  the  performance  of  the  entire  IADS. 

It  will  be  noted  that  the  three  levels  of  averaged  metrics  defined  above  are 
always  derivable  from  the  instantaneous  system  metric  (when  it  is  defined).  The 
instantaneous  system  metric  is  in  this  sense  an  attribute's  fundamental 
quantification,  of  which  the  various  averages  are  simply  higher-level  variants. 
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In  the  definitions  that  follow,  whenever  all  levels  of  metrics  are  applicable, 
explicit  formulations  of  the  instantaneous  system  metric,  the  time-averaged 
system  metric,  and  the  highest  level  average  -  the  IADS  or  roll-up  metric  -  are 
provided.  The  time-averaged  system  metric  may  be  regarded  as  an  intermediary 
between  the  fundamental  instantaneous  system  attribute  measure  and  the 
corresponding  roll-up.  Other  intermediate  averages,  including  the  instantaneous 
IADS  metric,  can  be  extracted  from  the  general  mathematical  procedure  for 
averaging  metrics,  as  outlined  in  Appendix  B  of  this  report.  In  the  event  that  the 
SIAP  attribute  measures  are  being  evaluated  from  Monte  Carlo  simulation  results 
with  multiple  runs,  the  instantaneous  system  metric  would  itself  normally  be 
computed  as  an  average  over  runs.  As  the  treatment  in  Appendix  B  shows,  run 
averages  can  also  be  incorporated  into  the  general  procedure,  with  essentially  no 
alteration  to  the  formalism. 

Data  supporting  the  SIAP  attribute  measures  are  assumed  to  be  available 
over  a  particular  time  period  of  interest,  which  would  normally  be  associated  with 
the  scenario  (the  CRS  if  the  evaluation  is  following  a  scripted  scenario).  For  the 
scoring  of  instantaneous  metrics,  the  time  period  of  interest  is  also  assumed  to 
be  subdivided  into  a  finite  number  of  scoring  subintervals.  This  division  may  be 
uniform  over  the  period  of  interest,  but  could  also  be  tailored  to  accommodate 
data  availability  or  experimental  design  factors.  Within  each  of  these 
subintervals,  a  scoring  time  is  selected  according  to  a  scheme  pre-established  by 
the  user  of  the  metrics.  Some  genera!  recommendations  for  establishing  scoring 
times  and  for  aligning  the  data  with  the  scoring  times  selected  are  provided  in 
SIAP  SE  TF  Technical  Report  2001-003. 

The  metrics  that  are  provided  for  each  attribute  below  are  in  no  way 
intended  to  limit  the  scope  of  any  analysis  (e.g.,  to  measure  the  performance  of 
systems,  perform  diagnostics,  or  find  root  causes  of  problems).  The  metrics  that 
are  specified  here  are  meant  to  provide  a  set  of  measures  that  are  consistently 
defined  between  analytic  efforts  and  can  be  used  to  share  and  compare  results. 
The  attributes  and  their  associated  metrics  are  to  be  considered  part  of  an 
evolutionary  process. 

The  computation  of  the  SIAP  attributes  requires  some  knowledge  of  truth 
regarding  objects.  In  the  absence  of  ground  truth,  such  as  in  the  case  of  an 
uninstrumented  live  exercise,  it  is  still  possible  to  obtain  valuable  estimates  of 
most  of  these  attributes  by  comparing  the  information  available  to  one  participant 
with  that  of  another,  or  by  examining  the  internal  consistency  of  each  individual 
participant’s  air  picture  over  time. 

The  following  notation  is  used  throughout  the  mathematical  definitions  in 
this  section: 

Atk  is  the  duration  of  the  k‘'^  time  subinterval  for  scoring; 

tk  is  the  k'*^  scoring  time; 

20 


4.2,3.1-SIAP  Attributes-TF-V2,0-0308 


K  is  the  total  number  of  scoring  times  over  the  time  period  of  interest  for 
the  scenario; 

M  is  the  total  number  of  participants  (or  units  to  be  scored)  in  the  scenario; 
J  is  the  total  number  of  objects  ever  occurring  in  the  scenario. 

In  the  case  of  a  uniform  division  of  the  time  period  of  interest  into  scoring 
subintervals,  Atk  =  (tend  -  tstart)/K  =  constant,  where  tstartand  tend  are  respectively 
the  start  and  end  times  for  evaluation,  but  in  fact  Atk  can  be  eliminated  from  every 
equation  in  this  section  when  it  is  constant.  It  need  not  be  assumed  that  either 
the  number  of  participants  or  the  number  of  objects  is  constant  over  time.  The 
constants  M  and  J  respectively  are  the  total  numbers  of  participants  and  objects 
that  are  ever  of  interest,  and  thus  serve  as  upper  bounds  for  the  relevant 
numbers  at  any  particular  time.  At  the  discretion  of  the  experiment  designer  or 
user  of  the  metrics,  a  participant  may  also  be  deemed  "not  of  interest"  at  times 
when  its  data  is  unavailable  due  to  experiment  design  limitations. 

Other  notation  is  defined  as  it  is  introduced  in  the  presentation  of  formulae 
for  the  metrics.  An  index  of  all  variables  used  will  be  provided  in  the  2003  update 
to  SIAP  SE  TF  Technical  Report  2001-003,  Appendix  A. 

2.3.1  TAMD  CRD-Retated  Attributes 

Attributes  discussed  in  this  section  correspond  to  the  SIAP  KPPs 
described  in  the  TAMD  CRD.  The  method  for  calculation  of  the  attribute 
measures  is  prescribed. 

2.3.1. 1  Completeness 

The  air  picture  is  complete  when  all  objects  are  detected,  tracked  and  reported. 

The  instantaneous  system  completeness  Cm(tk)  at  participant  m  at  the  k*^ 
scoring  time  tk  is: 


100% 


(1) 


where 

JTm(tk)  =  number  of  objects  with  at  least  one  assigned  track  held  by 
participant  m  at  time  tk,  and 

Jm(tk)  =  number  of  objects  at  time  tk,  assuming  that  participant  m  exists 
and  is  of  interest  for  scoring  at  time  tk.  If  participant  m  either  does  not 
exist  at  time  tk  or  is  not  to  be  included  in  the  scoring  of  the  metric  at  time  tk, 
then  both  JTm(tk)  and  Jm(tk)  are  set  to  zero  by  convention,  and  the 
instantaneous  system  completeness  is  not  defined  in  this  case.  (This 
latter  provision  applies  to  any  instantaneous,  participant-specific  quantity 
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entering  into  the  definition  of  any  SIAP  attribute  measure,  but  will  not 
hereafter  be  explicitly  noted.) 


The  time-averaged  system  level  measure  of  the  SIAP  completeness  Cm  is 
an  object-count  weighted  average  across  time; 


Z^TjtOAt, 

- 100%  (2) 

)Atk 


This  “object-count  weighted”  average  gives  more  weight  to  the  completeness 
during  periods  of  time  when  the  greatest  number  of  objects  are  present  in  the 
AOl.  Without  such  weighting,  the  attribute  could  be  dominated  by  its  value  during 
long  periods  when  there  were  very  few  objects.  (Similar  considerations  apply  to 
the  remaining  SIAP  attribute  measures.) 

The  roll-up  measure  of  the  SIAP  completeness  C  is  the  following  weighted 
average^  of  the  time-averaged  system  level  measure  across  participants; 

M  K 

ZEJTJUAt, 

- 100%,  0<C<100%.  (3) 

^  ^  )Atj^ 


The  roll-up  completeness  attribute  measure  C  can  range  in  value  from 
zero  to  100%.  The  goal  for  Completeness  is  established  by  the  Completeness 
KPP  in  the  TAMD  CRD.  (See  Appendix  A  for  a  discussion  of  the  relationship 
between  SIAP  Attributes  and  the  TAMD  and  CID  CRDs). 

2.3.1. 2  Clarity 

The  air  picture  is  clear  when  it  does  not  include  ambiguous  or  spurious 

tracks. 


The  quantification  of  the  clarity  attribute  comprises  two  measures;  one  for 
ambiguous  tracks  and  one  for  spurious  tracks.  Both  ambiguous  and  spurious 


'  While  the  sense  in  which  the  roll-up  measure  is  a  “weighted  average”  is  somewhat  abstract,  the 
averaging  formalism  presented  in  Appendix  B  of  this  report  shows  how  appropriate  weights  can 
be  defined  so  as  to  allow  interpretation  of  any  of  the  roll-up  metrics  in  this  report  as  weighted 
averages  -  of  either  time-averaged  system  metrics  or  instantaneous  IADS  metrics. 
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tracks  result  in  a  loss  of  clarity,  but  the  sources  of  the  errors  and  the  solution  sets 
for  the  errors  are  different. 

Ambiguous  Tracks 


Tracks  are  ambiguous  when  more  than  one  track,  assigned  to  the  same  object 
and  not  correlated  within  a  system,  are  displayable  to  a  participant. 


The  instantaneous  system  track  picture  ambiguity  Am(tk)  at  participant  m  at 
time  tk  is; 


A^(tk)  = 


NAjtJ 

JTjt,) 


(4) 


where  NAmCtk)  =  the  number  of  assigned,  uncorrelated  tracks  held  by  participant 
m  at  time  tk. 

The  time-averaged  system  level  measure  of  ambiguity  Am  is  a  tracked- 
object-count  weighted  average  across  time; 

XNA^(t,)At, 

- .  (5) 


The  roll-up  measure  of  overall  ambiguity  A  is  the  following  weighted 
average  of  the  time-averaged  system  level  measure  across  participants: 

M  K 

2lNAjk)Ak 

,  A>1,  (6) 

m=1  k=1 

The  Ambiguity,  A,  can  range  from  1  to  many  tracks  per  object.  The  air 
picture  may  be  said  to  be  perfectly  unambiguous  when  A  is  one.  The  goal  for  A  is 
established  by  the  TAMD  CRD  KPP  for  tracks  per  air  object. 

Spurious  Tracks 

A  track  is  spurious  when  it  is  not  assigned  to  any  object. 

The  instantaneous  system  measure  of  the  percentage  of  tracks  that  are 
spurious  Sm(tk)  as  measured  by  participant  m  at  time  tk  is; 
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(7) 


SJt,)  = 


Kik) 


100% 


where  Nm(tk)  =  the  number  of  tracks  held  by  participant  m  at  time  tk. 

The  time-averaged  system  level  measure  of  spurious  tracks  is  a  track- 
weighted  average  across  time: 

I[N„(t.)-NA„(k)lAk 

- K - 100%.  (8) 

k=1 

The  roll-up  measure  of  spurious  tracks  is  the  following  weighted  average 
of  the  time-averaged  system  level  measure  across  participants. 

M  K 

IZlNn.(tJ-NA„(k)]Ak 

s  =  . M . - 100% ,  0  <  S  <  100%.  (9) 

IZNJtjAt, 


In  practice  it  may  not  always  be  possible  to  measure  the  fraction  of 
spurious  tracks  in  exercises  because  this  attribute  requires  instrumented 
knowledge  of  all  objects  in  the  AOI.  When  collecting  data  in  a  live  exercise  with 
uncontrolled  commercial  air  traffic,  it  is  not  always  possible  to  determine  if  the 
tracks  were  generated  by  aircraft  or  were  truly  spurious.  However,  there  may  be 
features  of  the  systems  or  characteristics  of  some  tracks  which  may  identify  them 
as  spurious. 

The  percentage  of  tracks  that  are  spurious,  S,  ranges  in  value  from  0% 
when  perfect  to  100%.  The  TAMD  CRD  provides  a  KPP  for  the  percentage  of 
tracks  representing  distinct  objects.  Thus  the  CRD  requirement  bounds  the 
percentage  of  ail  tracks  that  are  not  spurious  (100%  -  S). 

Perfect  clarity  is  attained  when  the  air  picture  is  perfectly  unambiguous 
(A=1)  and  there  are  no  spurious  tracks  (S=0). 

2.3. 1.3  Continuity 

The  air  picture  is  continuous  when  the  track  number  assigned  to  an  object  does 
not  change. 

The  air  picture  at  any  given  time  is  represented  by  a  collection  of  track 
reports.  Continuity  is  the  measure  of  the  persistence  and  consistency  of  these 
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reports  over  time.  The  track  number  assigned  to  an  object  is  the  alphanumeric 
"tag"  by  which  the  track  assigned  to  the  object  is  identified  (cf.  MIL-STD-601 1B, 
1999).  The  amplifying  data  on  the  object  with  which  a  track  is  associated  is  a  set 
of  information  unique  to  the  track  number.  A  loss  or  change  of  track  number  may 
entail  loss  or  change  of  information  on  the  object  associated  with  the  track  report. 
The  track  number  will  change  if  the  track  is  dropped  and  later  picked  up  with  a 
different  track  report,  or  if  tracks  are  swapped  between  two  objects  that  have 
been  confused.  The  track  number  may  also  change  when  ambiguous  tracks  are 
resolved. 

The  SIAP  attribute  of  continuity  addresses  only  those  aspects  of 
information  consistency  which  are  directly  associated  with  the  track  number.  The 
more  general  notion  of  information  continuity  stretches  across  this  attribute,  the 
SIAP  ID  attributes  (discussed  below)  and,  in  the  case  of  information  specific  to 
particular  tracks  and  objects,  certain  SIAP  MOPs  (cf.  SIAP  Technical  Report 
2001-002). 

For  SIAP  assessments,  the  attribute  of  continuity  is  quantified  in  two  ways. 
The  first  is  the  Rate  of  Track  Number  Changes  and  the  other  the  Longest  Track 
Segment. 

The  Rate  of  Track  Number  Changes  on  object  j  from  the  perspective  of 
participant  m  is; 


(10) 


where 

TTj,m=  total  time  object)  is  tracked  by  participant  m, 

NUj,m  =  the  minimum  number  of  track  numbers  assigned  to  object  j  which 
cover  the  period  during  which  object)  is  tracked  by  participant  m. 
Mathematical  definitions  of  the  variables  Tii.^and  NUj.m ,  further  broken 
down  in  terms  of  track-to-truth  assignment  results,  can  be  found  in 
Appendix  A  of  SIAP  SE  TF  Technical  Report  2001-003. 

The  concept  embodied  in  this  definition  is  somewhat  more  abstract  than  other 
properties  represented  by  the  SIAP  attributes,  and  the  visualization  provided  by 
Figures  4  and  5  may  help  to  clarify  the  idea. 


25 


4.2.3. 1  -SIAP  Attributes-TF-V2. 0-0308 


Object] 


Figure  4.  Example  of  determining  Characteristic  T rack  Lifetime 

Time  the  object  is  tracked  (Tij  ^)  is  the  sum  of  the  double  (blue)  lines.  The 
minimum  set  of  tracks  that  cover  the  period  the  object  is  tracked  consists  of: 
TN101,  TN202,  TN302,  and  TN303  (indicated  by  the  boxed  areas).  The  minimum 
number  of  tracks  used  to  determine  the  rate  of  track  number  changes  for  this 
example  is  4. 

This  definition  is  not  affected  by  the  problem  posed  by  a  track  assignment 
algorithm  where  the  assignment  algorithm  jumps  back  and  forth  between  two 
possible  track  assignments  (see  Figure  5). 


Figure  5.  Multiple  assignments  of  a  track  to  two  track  numbers 
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In  Figure  5,  the  double  (blue)  tine  represents  the  time  the  object  is  tracked.  The 
tracks  that  cover  the  period  when  the  object  is  tracked  are  TN  101  and  TN  201 . 
The  minimum  number  of  tracks  for  this  example  is  2.  A  different  track 
assignment  algorithm  might  have  made  different  choices,  but  that  will  not  affect 
the  rate  of  track  number  changes.  No  matter  how  fast  the  track  assignment 
algorithm  swaps  the  assignment  during  the  time  the  two  objects  are  too  close  to 
distinguish  accurately  what  is  correct,  the  number  of  tracks  assigned  to  the  object 
will  remain  the  same. 

The  system  based  Rate  of  Track  Number  Changes  (for  participant  m) 
would  be  the  object  weighted  average: 

-  ■  (11) 

in™ 


The  IADS  roll-up  of  the  Rate  of  Track  Number  Changes  is  the  following 
weighted  average  across  participants: 

M  J 

XI(NU,„-1) 

R  =  - .  (12) 

SITT,,. 

nrtsti  1=1 


The  Longest  Track  Segment  is  a  measure  of  the  longest  time  a  track  is 
assigned  to  an  object,  relative  to  the  time  the  object  is  in  the  AOI.  Expressed  as 
a  percentage  of  time,  the  Longest  Track  Segment  LSm  at  participant  m  is: 


LS,,=£L - 100%  (13) 

IT 

j=i 


where 

TLm.j  =  time  duration  of  the  longest  continuous  single  track  segment  with 
the  same  track  number  held  by  participant  m  assigned  to  object  j, 

Tj  =  total  time  of  flight  for  object  j,  and 
J  =  the  number  of  objects  over  the  evaluation  period. 

The  roll-up  measure  of  the  Longest  Track  Segment  LS  is  the  following 
weighted  average  of  the  above  over  participants: 

27 


4.2.3. 1-SIAP  Attributes-TF-V2.0-0308 


0<LS<100%. 


(14) 


M  J 


LS  =  =lft - 100%, 

ZIT, 

m~-1  j=1 


When  LS  equals  100  percent,  all  of  the  participants  hold  a  stable, 
continuous  track  file  on  each  object.  This  does  not  preclude  spurious  or 
redundant  tracks,  and  these,  if  they  do  occur,  need  not  be  continuous.  If 
Completeness  (C)  is  not  100  percent  (so  that  there  are  untracked  objects),  then 
LS  cannot  take  a  value  larger  than  C. 

In  the  limit  of  perfect  Continuity,  the  Longest  Track  Segment  (LS) 
approaches  the  value  of  Completeness  (C),  and  the  rate  of  track  number 
changes  approaches  zero. 

2,3. 1.4  Kinematic  Accuracy 

The  air  picture  is  kinematically  accurate  when  the  position  and  velocity  of  each 
assigned  track  agree  with  the  position  and  velocity  of  the  associated  object. 

Different  tasks  have  different  requirements  for  SIAP  accuracy.  For  general 
background  objects  for  which  participants  have  a  low  level  of  interest  and  no 
engagements  are  underway,  the  objective  is  to  maintain  SA.  For  the  purpose  of 
SA,  the  track  accuracy  needs  to  be  good  enough  for  correlation  with  local  tracks 
so  that  they  are  not  dualed  or  misidentified. 

Once  an  engagement  has  been  ordered,  the  engagement  target  and  all 
other  objects  in  the  vicinity  will  require  special  attention.  Track  accuracy 
requirements  increase  in  order  to  support  the  engagement,  maintain  SA,  cue  fire 
control  radar,  vector  fighters,  and  guide  missiles  in  flight.  A  different  level  of  track 
accuracy  is  required  by  the  engaging  units.  When  engaging  units  are  integrated 
for  the  purpose  of  fire  control,  such  that  one  unit  fires  on  the  basis  of  another’s 
sensor  data,  the  highest  level  of  track  accuracy  is  required.  In  general  the 
kinematic  accuracy  of  the  track  is  going  to  depend  on  the  activity  and  the  special 
needs  of  the  participant. 

For  the  purpose  of  assessing  the  adequacy  of  the  SIAP  to  support  these 
different  levels  of  activity,  a  measure  of  the  absolute  track  accuracy  is  needed.  A 
mathematical  definition  of  the  position  and  velocity  accuracy,  PAj,m(tk)  and 
VAj,m(tk),  for  object  j  tracked  by  participant  m  at  time  tk,  is  provided  by  separately 
taking  root  mean  square  (RMS)  averages  of  the  track  position  and  velocity  errors 
over  all  tracks  assigned  (to  that  object,  at  that  time  for  that  participant): 
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PA,,m(t)  =  ,|i2=!y - — — -  (15) 

NAj.  m(tj^  ) 


^~|V)^.n,m(t|^  +  VYj,n,m{t|^  +  W\^,am(t|^ 

VA,^t)  =  ,|^S5idy - -  (16) 


where 

Xj,n,m{tk),  Yj,n,m(tk),  Zj,n,m(tk)  are  the  Cartesian  position  coordinates  for  object 
j  in  the  local  east-north-up  coordinate  frame  of  track  n  held  by  participant 
m  (track-centered  coordinate  frame)  at  time  tk, 

VXj,n.m(tk).  VYj,n,ni(tk),  VZj,f,,m(tk)  are  the  velocity  components  of  object  j  in  the 
local  east-north-up  coordinate  frame  of  track  n  held  by  participant  m 
(track-centered  coordinate  frame)  at  time  tk, 

Dj,m(tk)  is  the  set  of  tracks  held  by  participant  m  and  assigned  to  object)  at 
time  tk, 

NAj,tn(t)  is  the  cardinality  of  Dj,m(tk),  that  is,  the  number  of  tracks  held  by 
participant  m  that  are  assigned  to  object  j  at  time  tk,  and 
w  is  a  weighting  used  to  reduce  the  significance  of  the  vertical  position 
and  velocity  errors  on  2-D  sensors.  Currently  all  SIAP  assessments  are 
using  constant  weight  values  of  either  one  or  zero  over  all  participants 
considered  in  the  assessment,  but  special  considerations  for  complex 
architectures  with  mixes  of  2-D  and  3-D  sensors  may  be  explored  in  the 
future. 

The  instantaneous  system  level  measures  of  position  and  velocity 
accuracy,  PAm(tk)  and  VAm(tk),  are  respectively  the  RMS  averaged  position  and 
velocity  errors  over  all  assigned  tracks  held  by  participant  m  at  time  tk: 


PKik)  = 


i=i 


I  ENAj,rn(t,) 


(17) 


VA,,(tJ  = 


j=i 


^NA|,m(t,,  ) 


(18) 
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Note  that  weighting  the  sums  over  objects  by  NAj_m(t)  has  the  same  effect  as 
taking  an  unweighted  average  over  assigned  tracks  (see  Appendix  B). 

The  time-averaged  system  level  measures  of  position  and  velocity 
accuracy,  PAm  and  VAm,  are  the  analogous  RMS  averages  across  assigned 
tracks  and  scoring  times: 


K  J 


PA. 


EZNAj,m(tkFAj^(t,f  At, 

k=1  j=1 


K  J 


ZZNA,„(t.)At, 

k=1  j=1 


(19) 


(20) 


Finally,  the  roll-up  measures  are  the  analogously  weighted  averages 
across  participants  of  the  time-averaged  system  metrics: 


M  K  J 

X  ZENAi,n.(t,)PAj,Jt,)^At, 

PA=  . - ,  PA>0,  (21) 

I  E  EENA,,jt,)At, 

}  m=1  k=1  j=1 


VA>0. 


(22) 


In  general,  the  requirement  for  accuracy  depends  on  the  needs  of  the 
user.  Although  the  kinematic  accuracy  provides  a  necessary  diagnostic,  any 
requirement  for  accuracy  depends  on  the  mission.  For  the  mission  of  achieving 
and  maintaining  a  consistent  and  coherent  air  picture,  the  accuracy  must  be 
sufficient  to  permit  the  correct  correlation  of  the  network  track  with  the  local  track 
and  avoid  miscorrelation  of  local  tracks  with  the  wrong  network  track.  Kinematic 
accuracy  is  therefore  a  contributing  factor  in  the  SIAP  attributes  of  Clarity  and 
Continuity.  In  addition,  kinematic  accuracy  is  the  central  factor  in  determining 
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whether  or  not  closely  spaced  objects  (relative  to  the  resolution  of  the  sensors) 
may  swap  track  numbers  and  CIDs  from  one  to  the  other. 

The  TAMD  CRD  (2001)  does  not  provide  a  single  KPP  requirement  for 
Accuracy  although  it  does  attempt  to  establish  a  limit  to  the  error  introduced  in 
the  position  of  a  track  due  to  latency  in  the  network.  This  is,  of  course,  just  one 
part  of  the  overall  contribution  to  the  kinematic  accuracy  -  other  factors  include 
failure  to  achieve  time  synchronization,  poor  data  registration  and  gridlock, 
deficiencies  In  processing  by  tracking  or  correlation  algorithms,  to  name  a  few. 
The  Draft  TAMD  CRD  that  is  currently  in  the  JROC  review  cycle  does  specifies 
track  accuracy  requirements  for  both  position  and  velocity,  which  correspond  to 
the  SIAP  attribute  measures  of  Position  Accuracy  and  Velocity  Accuracy. 

2.3.1, 5  Discusston  on  Timeliness 

"Timeliness"  is  mentioned  as  a  required  SIAP  attribute  in  the  TAMD  CRD. 
However,  the  CRD  offers  neither  a  general  definition  of  SIAP  timeliness  nor  a 
standard  for  the  timeliness  of  track  data.  For  each  top-level  Information 
Exchange  Requirement  (lER)  in  the  CRD,  including  for  track  data,  timeliness  is 
"to  be  determined." 

Although  there  will  be  no  explicit  determination  of  "timeliness”  per  se  at 
the  level  of  SIAP  attributes,  the  SIAP  SE  TF  recognizes  the  importance  of  the 
timeliness  requirement,  and  will  recommend  the  use  of  various  measures  of 
performance  (MOPs)  to  capture  the  effects  of  network  time  delays  in  root  cause 
analysis.  A  typical  MOP  might  be  the  elapsed  time  between  the  availability  of 
certain  information  for  distribution  on  a  network  and  the  time  that  the  information 
is  available  to  a  participant  who  needs  it.  Examples  of  effects  (expected  to  be 
second-order)  on  quantified  SIAP  attributes  would  be:  the  degradation  of 
completeness  due  to  undistributed  information  during  the  elapsed  time  period, 
the  degradation  of  clarity  from  dealing  and  iocal/remote  miscorrelation  (which  can 
be  exacerbated  by  unaccounted-for  time  lags),  dropped  tracks  (reflected  by  a 
decline  in  continuity),  decay  in  accuracy  overtime  (e.g.,  latency  issues 
associated  with  I FC  accuracy  requirements),  and  non-commonality  due  to  non- 
uniform  distribution  of  data. 

Timeliness  enters  into  higher-level  warfighting  benefits  assessments  as 
well,  and  other  metrics  would  arise  from  analysis  of  any  of  the  instantaneous 
attribute  measures  in  conjunction  with  the  decision-making  or  engagement 
timeline  for  a  particular  mission  (e.g.,  does  a  decision-maker  or  engaging  unit 
have  the  quality  of  information  needed  by  the  time  the  decision  must  be  made  or 
the  engagement  executed?).  The  SIAP  SE  TF  will  address  this  aspect  of 
timeliness  through  measures  of  effectiveness  (MOEs)  at  the  mission,  force  and 
campaign  levels  of  the  metrics  hierarchy. 
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With  regard  to  CID  timeliness,  the  CID  CRD  does  specify  a  standard  KPP 
for  the  timeliness  of  CID  data. 

“The  CID  process  must  be  accomplished: 

a.  Soon  enough  to  provide  military  options  including  warning  and 
engagement  decisions. 

b.  Without  increasing  the  engagement  time  for  any  weapon 
system  more  than  XX  seconds.” 

The  SIAP  SE  TF  has  not  assigned  a  SIAP  attribute  that  corresponds  to 
this  KPP.  However,  as  already  suggested,  the  assessment  of  warfighting  benefits 
will  proceed  within  the  context  of  specific  missions,  for  which  MOEs  will  be 
introduced  to  cover  this  KPP. 


2.3.2  CID  CRD“Related  Attributes 

The  CID  attributes  are  completeness,  correctness,  and  clarity.  The  CID 
attributes  of  completeness  and  correctness  address  the  KPPs  in  the  CID  CRD. 
The  clarity  attribute  is  introduced  to  provide  an  explicit  measure  of  ID  consistency 
across  the  TAMD  FoS  in  the  case  where  there  are  multiple  tracks  on  an  object. 

The  CID  attributes  are  assessed  separately  for  the  sets  of:  friendly  (Blue) 
truth  objects,  hostile  (Red)  truth  objects,  and  neutral  or  other  truth  objects  if 
applicable.  An  additional  measure  is  computed  over  objects  of  all  allegiance 
classes.  Each  CID  attribute  thus  yields  a  separate  score  for  each  of  the  three 
force  allegiance  classes,  plus  an  overall  value.  The  CID  attributes  are  based  on 
the  classification  of  a  participant’s  instantaneous  track  data  on  a  particular 
tracked  object  into  one  of  four  different  ID  states: 

•  Correct 

•  Not  Identifed 

•  Incorrect 

•  Ambiguous 

The  ID  state  of  an  object  is  correct  if  at  least  one  associated  track  is  identified 
correctly  and  no  associated  tracks  are  identified  incorrectly.  The  ID  state  of  an 
object  is  incorrect  if  at  least  one  associated  track  is  identified  incorrectly  and  no 
associated  tracks  are  identified  correctly.  An  object’s  ID  state  is  ambiguous  if  at 
least  one  associated  track  is  identified  correctly  and  at  least  one  associated  track 
is  identified  incorrectly.  The  ID  state  of  an  object  is  not  identified  if  ail  associated 
tracks  are  either  unknown  or  pending.  Since  these  four  states  are  mutually 
exclusive,  exhaustive,  and  account  for  all  tracked  objects,  the  fractions  of  tracked 
objects  falling  into  each  of  the  four  states  sum  to  unity.  Three  independent 
measures  are  thus  defined.  The  attributes  of  ID  completeness,  correctness,  and 
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clarity,  presented  below,  correspond  to  one  possible  way  of  selecting  three 
independent  measures. 

Table  1  describes  an  example  set  of  rules  for  assessing  the  correctness  of 
a  CID  label  on  an  object.  The  ID  taxonomy  (first  column  of  Table  1}  is  taken  from 
MIL-STD-2525B  (1999).  For  objects  of  each  force  allegiance  type  (Friend, 

Hostile  or  Neutral),  the  table  provides  a  possible  definition  of  "Correct," 

"Incorrect,”  and  "Not  Identified."  The  definition  used  depends  upon  the  scenario 
and  judgement  of  the  Force  Commander.  The  CRS  used  by  the  SIAP  SE  TF  will 
include  a  definition  of  correct,  incorrect,  unknown,  and  ambiguous.  By  no  means 
are  the  assignments  shown  in  Table  1  intended  to  limit  CID  designations  to  the 
given  set;  in  particular,  this  is  not  meant  to  imply  that  ID  "correctness"  should 
necessarily  be  limited  to  force  allegiance  considerations.  The  sample 
assignments  are  simply  given,  in  lieu  of  a  complete  scenario  description,  to  help 
illustrate  the  attributes  associated  with  CID  in  the  discussion  to  follow. 


Table  1 

Example  Set  of  CID  Designations 


ID  of  Truth  Ob 

ect 

ID  of  Track 

Friend 

Hostile 

Neutral 

Pending 

N 

N 

N 

Unknown 

N 

N 

N 

Assumed  Friend 

C 

1 

1 

Friend 

C 

1 

1 

Neutral 

1 

1 

C 

Suspect 

1 

C 

1 

Hostile 

1 

C 

1 

C  =  Correct  N  =  Not  Identified  I  =  Incorrect 

The  next  three  attributes  are  specifically  associated  with  the  CID  CRD 
KPPs.  These  are  ID  completeness,  ID  correctness,  and  ID  clarity. 
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2.3.2.1  ID  Completeness 


The  ID  is  complete  when  all  tracked  objects  are  in  an  identified  state  (i.e.,  a  state 
other  than  “not  identified”). 

The  instantaneous  system  ID  completeness  can  be  represented  by  the 
percentage  of  objects  CiDx„(tk)  of  allegiance  type  X  at  participant  m  at  time  tk  that 
are  complete: 


where 


(tk) 


J 


100% 


(23) 


JTXm(tk)  =  the  number  of  tracked  objects  of  allegiance  type  X  tracked  by 
participant  m  at  time  tk,  the  type  variable  X  taking  the  possible  values  B 
(Blue),  R  (Red)  or  N  (Neutral), 

JuXn,(tk)  =  the  number  of  tracked  objects  of  allegiance  type  X  in  the  “not 
identified”  ID  state  held  by  participant  m  at  time  tk.  (This  includes  any 
specific  ID  assignments,  or  combinations  thereof,  that  are  defined  to  be 
"not  identified"  assessments  in  the  CRS.) 


The  corresponding  overall  measure  of  ID  completeness  Cid  (tk)  may  be 

computed  from  the  three  allegiance  type-specific  measures  and  the  tracked 
object  counts,  as  follows: 


(^k )  ■ 


(t,  )C„_  (t,  )  +  JTR  Jt,  )C„,_  (t, )  +  jTN^(t^  (t, ) 


JT  (t  1 

m  k 


(24) 


or  equivalently,  equation  23  may  be  applied  with  the  ‘X’s  omitted  and  with  no 
consideration  of  force  allegiance  in  the  variable  definitions. 

The  time-averaged  system  level  measures  of  the  CID  completeness  Cidx 
are  the  object-weighted  averages  of  the  instantaneous  measures  over  time: 


X[JTX,(tk)-JUX^(tJ]At, 

^ - 100%,  (25) 

XJTXjtjAt, 

k=1 


where  the  ‘X’  is  omitted  if  the  overall  measure  is  being  computed. 

The  roll-up  measure  Cidx  is  the  analogously  weighted  average  across 
participants: 
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M  K 

C,DX  =  — - m'  ■  K - 1 00%,  0  <  C,ox  <  1 00%.  (26) 

m=1  k=1 


where  again,  the  ‘X’  is  omitted  if  the  overall  ID  completeness  roll-up  CiD  is  to  be 
computed  (this  stipulation  applies  to  the  overall  measures  for  the  remaining  CID 
attributes  as  well,  but  will  not  be  explicitly  noted  hereafter.) 

The  fraction  of  ID  completeness  ranges  from  zero  to  100%.  The  CiD  CRD 
establishes  a  requirement  for  the  completeness  (CID  Probability  of  ID  KPP)  of 
the  CID  information  broken  down  by  object  force  allegiance  -  hostile,  friend,  and 
neutral.  Hence  the  completeness  goals  in  the  CRD  equate  to  thresholds  and 
objectives  for  the  attribute  measures  Cidr,  Cidb,  and  Cidn. 

2.3.2.2  ID  Correctness 

The  ID  is  correct  when  all  tracked  objects  are  in  the  correct  ID  state. 

The  instantaneous  system  ID  correctness  can  be  represented  by  the 
percentage  of  objects  IDcXm(tk)  of  allegiance  type  X  labeled  correctly  at  a 
participant  m  at  time  tk; 


iDcx„(t,)=^!2M!ilioo% 

'  XmOk  / 


(27) 


where 

JcXm(tk)  =  number  of  tracked  objects  of  allegiance  type  X  in  the  correct  ID 
state  (as  defined  in  the  CRS)  held  by  participant  m  at  time  tk. 

The  corresponding  overall  measure  of  ID  correctness  IDc^(tk)  may  be  computed 

from  the  three  allegiance  type-specific  measures  and  the  tracked  object  counts, 
as  follows: 


)1D,3^  (t, )  + JTR  Jt,  )IDe«^  (t,  )  +  JTN^  (t,  jlD^.^  (t,  ) 


Jt  (t  ) 

m ''  k  ^ 


(28) 


or  equivalently,  equation  27  may  be  applied  with  the  ‘X’s  omitted  and  with  no 
consideration  of  force  allegiance  in  the  variable  definitions. 

The  time-averaged  system  measure  of  the  ID  correctness  IDcx^  is  the 
object-weighted  average  of  the  above  over  time: 
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(29) 


XJCXjtjAt, 

IDCX^  =  - 100%. 

i;jTX^{t,)At, 


The  roll-up  measure  IDcx  is  the  analogously  weighted  average  of  this 
across  participants: 


IDcx  = 


M  K 

S  {‘ic 


m=1  k»-.1 
M  K 


ZZJTX,(t,)At, 


100%, 


0<  IDcx  <100%.  (30) 


The  range  of  possible  values  for  ID  correctness  is  from  zero  to  100%.  The 
CID  CRD  establishes  a  KPP  requirement  on  the  conditional  probability  of  correct 
identification  of  an  object  given  that  the  object  is  “characterized  by  CID  systems” 
(CID  CRD,  2001).  Interpreting  “characterized”  as  “in  a  state  other  than  ‘Not 
Identified’”  (an  interpretation  consistent  with  the  KPP  language  and 
accompanying  examples  in  the  CID  CRD),  this  translates  to  a  requirement  on  the 
ratio  of  overall  ID  correctness  to  overall  ID  completeness,  IDc/ClD. 

2.3.2.3  ID  Clarity 

The  ID  is  clear  if  no  tracked  object  is  in  the  ambiguous  ID  state. 

The  instantaneous  system  ambiguity  IDAXm(tic)  of  the  CID  for  objects  of 
allegiance  type  X  at  participant  m  at  time  tk  is  given  by; 

IDAX„,(t,)  =  — ^NoO%  (31) 

JTXjtJ  ^  ^ 

where 

jAXm(tk)  =  number  of  tracked  objects  of  allegiance  type  X  in  the  ambiguous  ID 
state  held  by  participant  m  at  time  tk- 

The  corresponding  overall  measure  of  ID  ambiguity  IDAjtk)  may  be  computed 
from  the  three  allegiance  type-specific  measures  and  the  tracked  object  counts, 
as  follows: 


iDa  (tk)^ 


(t,  )ID^,^  (tk  )  +  JTR^  (tk  )ID^,^  (tk )  +  JTN^  (tk  )ID,,^  (tk ) 


JT  {t  ) 


(32) 


or  equivalently,  equation  31  may  be  applied  with  the  ‘X’s  omitted  and  with  no 
consideration  of  force  allegiance  in  the  variable  definitions. 
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The  time-averaged  system  level  measure  of  CID  ambiguity  IDAXm  is  the 
object-weighted  average  of  the  above  over  time; 

|;jAX,(tjAt, 

IDax^  =  - 100%.  {33} 

^JTXjtjAt, 

k=1 

The  roll-up  measure  IDax  is  the  analogously  weighted  average  of  this 
across  participants: 


IDax  = 


M  K 


m=1  k=1 


M  K 


m~'1  k=1 


100%, 


0<  IDAX  <100%.  (34) 


The  possible  range  of  values  for  ID  ambiguity  is  from  zero  to  100%.  As 
already  noted,  there  is  no  CID  CRD  KPP  currently  associated  with  this  attribute. 
Clearly  the  objective  of  the  SIAP  is  to  keep  the  ambiguity  as  close  to  zero  as 
possible. 

Because  the  fractions  corresponding  to  the  four  ID  states  sum  to  unity,  an 
attribute  defining  ID  Incorrectness  is  redundant. 

2.3.3  Combination  Attribute 

According  to  the  TAMD  CRD  (2001),  “The  SIAP  consists  of  common, 
continuous,  and  unambiguous  tracks  of  airborne  objects  of  interest  in  the 
surveillance  area.”  “Common”  in  this  context  is  generally  understood  to  involve  a 
consistent  understanding  among  all  participants.  While  it  is  not  intended  that  all 
network  participants  must  receive  all  data,  it  is  desired  that  any  information 
needed  by  a  participant  or  a  group  of  participants  be  available  to  them.  Therefore 
if  a  group  of  participants  have  a  need  for  the  same  information,  it  should  be  held 
by  all  of  them.  Information  needed  for  SA  is  broadly  needed  by  all  network 
participants;  thus,  generally  speaking,  the  SA  picture  should  be  consistent 
among  all  participants. 

2.3.3.1  Commonality 

The  air  picture  is  common  when  the  assigned  tracks  held  by  each  participant 
have  the  same  track  number,  position,  and  ID. 
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The  instantaneous  commonality  CM(ti<)  is  computed  as  follows; 


CM(t )  =  1 00%  (35) 

'  Ns(t,)  ^  ^ 

where 

Nc(tk)  =  the  number  of  assigned  tracks  held  by  all  participants  at  time  tk, 
such  that: 

-each  track  is  represented  by  a  unique  track  number,  common  to  all 
participants 

-position  and  time  data  on  each  track  is  the  same  for  all  participants,  to 
within  specified  tolerances  (defaults  are  currently  ±  5  min  in  latitude  and 
longitude,  ±  20  kft  in  altitude) 

-associated  ID  data  on  each  track  is  the  same  for  all  participants 
and 

Ns(tk)  =  the  number  of  assigned  tracks  held  by  at  least  one  participant  at 
time  tk- 


time: 


The  roll-up  measure  Cm  is  obtained  by  track-weighted  averaging  over 


^NcCtJAt, 
CM  =  ¥ - 


ZNS(t,)Ak 

k=1 


100%,  0<Cm<100%. 


(36) 


The  computation  of  the  roll-up  of  commonality  finds  the  track  information  that  is 
held  simultaneously  by  all  participants.  As  the  number  of  participants  goes  up, 
the  commonality  of  the  network  participants  will  typically  go  down.  For  some 
diagnostic  purposes  it  may  be  useful  to  compute  the  commonality  on  a  pair-wise 
basis.  That  is,  compare  the  track  stores  of  each  participant  with  every  other 
participant.  This  calculation  will  determine  if  any  lack  of  commonality  seen  in  the 
roll-up  is  attributable  to  a  particular  participant  or  if  it  was  a  generalized 
deficiency.  An  alternative  to  the  roll-up  in  equation  36  would  be  the  average  over 
all  participants  of  the  pair-wise  commonality  of  each  participant  with  all  possible 
partners.  This  definition  may  be  found  to  be  useful,  and  if  so,  the  definition  of 
commonality  could  evolve. 

2.3.4  Formation  Tracking  Considerations 

As  noted  in  the  Introduction  to  this  report,  the  definitions  and  formulae  of 
Sections  2.1  through  2.3.3. 1  are  framed  to  support  published  SIAP  and  CID 
requirements  -  specifically  those  expressed  in  the  KPP  language  of  the  TAMD 
CRD  (2001)  and  the  CID  CRD  (2001).  These  2001  documents  make  no  explicit 
reference  to  the  use  of  formation  tracks.  A  formation  track  is  a  track  intentionally 
representing  two  or  more  objects,  the  number  of  objects  being  designated  by  an 
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amplifying  data  element  known  as  the  track  strength.  Since  requirements  in  the 
TAMD  CRD  (2001)  take  no  account  of  formation  tracks,  the  SIAP  KPP  has  in  the 
past  been  interpreted  as  imposing  a  strict  “one  track  per  object”  condition  on  the 
SIAP,  without  regard  for  any  use  of  formation  tracking  or  the  track  strength  field 
(CINCUSJFCOM  J8  Memorandum,  21  June  2001). 

However,  the  SIAP  SE,  JTAMDO,  USJFCOM,  and  other  parties  to  the 
current  TAMD  CRD  revision  process  are  now  advocating  a  revision  to  the  SIAP 
requirements  that  would  explicitly  acknowledge  the  use  of  formation  tracking  as  a 
means  to  attain  a  SIAP  (TAMD  CRD,  March  2003  draft).  (Indeed,  the  proposed 
language  would  require  the  use  of  formation  tracks  in  certain  circumstances, 
though  this  requirement  would  not  be  part  of  the  SIAP  KPP.)  The  revised  TAMD 
CRD  has  not  yet  been  validated  by  the  JROC,  but  there  is  no  reason  to  suppose 
that  the  JROC  would  reject  the  amended  SIAP  requirements. 

It  is  the  intent  of  the  SIAP  SE  to  adapt  the  attributes  defined  in  this  report 
to  accommodate  the  use  of  formation  tracks  in  accordance  with  the  next 
published  revision  of  the  TAMD  CRD.  For  some  of  the  SIAP  attribute  measures, 
this  will  be  fairly  straightforward,  at  least  when  the  track  strength  is  reported  as  a 
number.  For  instance,  if  it  is  assumed  that  every  track  of  strength  n  is  eligible  for 
assignment  to  up  to  n  truth  objects  at  a  given  time,  the  Completeness  measure 
(Section  2.3. 1.1)  can  be  computed  by  equation  1  with  the  variable  JTm(tk) 
interpreted  as  the  number  of  objects  covered  by  these  (possibly  one-to-many) 
assignments  (at  participant  m,  time  tk).  The  Clarity  measures  (Section  2. 3. 1.2) 
can  be  similarly  adjusted  by  reinterpreting  the  variables  NAm(tk)  and  Nm(tk)  as. 
respectively,  the  sum  of  the  strengths  of  assigned  tracks  and  the  sum  of  the 
strengths  of  all  tracks  (held  by  participant  m  at  time  tk).  However,  the  details  of 
this  one-to-many  assignment  procedure  have  yet  to  be  worked  out  in  a  form 
suitable  for  automated  implementation.  Furthermore,  there  are  more  subtle 
effects  on  some  of  the  other  of  the  other  SIAP  attributes  (continuity  and 
accuracy,  for  example)  which  have  not  been  addressed.  A  complete  schema  for 
accommodating  formation  tracks  within  the  SIAP  assessment  methodology  wilt 
therefore  be  deferred  to  the  2004  revisions  of  SIAP  SE  Technical  Reports  2001- 
001  and  2001-003. 


3.  CONCLUSIONS  AND  RECOMMENDATIONS 

Meaningful  and  precise  definitions  and  derivations  of  SIAP  attributes  have 
been  presented,  based  upon  the  guidelines  found  in  the  TAMD  and  CID  CRDs. 
These  definitions  are  quantifiable,  testable  and  measurable.  Having  defined 
these  attributes,  the  SIAP  SE  TF  will  use  them  as  part  of  a  system  engineering 
process  to  successfully  evaluate,  predict,  and  prescribe  meaningful  engineering 
recommendations  for  a  SIAP.  With  universal  SIAP  attribute  definitions  effectively 
vetted  through  the  services  and  applicable  agencies,  the  joint  community  will 
have  a  common  reference  for  defining  SIAP-related  warfighting  capability. 
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Analysis  will  be  conducted  using  more  than  one  vignette  and  more  than 
one  scenario  to  ensure  a  wide  variety  of  tactical  circumstances.  This  will  help 
eliminate  scenario  dependencies  and  permit  more  comprehensive  analysis.  A 
high-level  description  of  scenarios  currently  under  consideration  is  provided  in 
SIAP  SE  TF  Technical  Report  2002-003.  The  details  of  these  campaigns  are  to 
be  provided  in  other  documents,  for  reasons  of  classification. 

The  companion  report  SIAP  SE  TF  Technical  Report  2001-002  outlines 
the  plan  for  incorporation  of  the  SIAP  attributes  into  an  overall  hierarchy  of  SIAP 
metrics,  including  MOPs  and  MOEs.  Various  SIAP  SE-sponsored  analysis 
efforts  (some  underway,  others  still  in  planning)  are  aimed  at  providing 
quantitative  traceability  among  selected  metrics  at  different  levels  in  this 
hierarchy  (see  Figure  6).  The  companion  report  SIAP  SE  TF  Technical  Report 
2001-003  provides  procedures  for  implementing  evaluation  of  SIAP  attribute 
measures  in  an  automated  data  collection  and  processing  environment.  The 
SIAP  SE  TF  plans  to  publish  updates  to  all  three  of  these  documents 
approximately  annually.  While  additional  refinement  to  the  definitions  of  metrics 
is  still  needed  at  the  MOP  level,  and  the  automated  implementation  plan  is 
expected  to  further  evolve,  the  definitions  and  supporting  mathematics  specific  to 
the  SIAP  attributes  as  presented  in  this  report  are  expected  to  remain  relatively 
constant  over  time. 


Figure  6.  SIAP  attributes  within  the  traceability  process 
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APPENDIX  A. 


Correspondence  of  TAMD  and  CID  CRD  Requirements  to  SIAP  Attributes 

The  JROC  has  already  established  some  standards  for  assessing  a  SIAP 
in  the  form  of  Key  Performance  Parameters  (KPPs)  in  the  TAMD  and  CID  CRDs. 
It  is  taken  as  a  given  that  these  must  be  reflected  within  the  final  set  of  SIAP 
Attributes.  In  defining  the  SIAP  Attributes,  the  SIAP  SE  TF  has  found  it 
necessary  to  augment  the  KPPs  slightly  to  arrive  at  a  complete  description  of  all 
aspects  of  the  SIAP  believed  to  be  both  independent  and  significant  for  analysis 
purposes,  in  addition,  since  the  CRD  KPPs  are  oriented  towards  requirements 
rather  than  analysis,  the  SIAP  SE  TF  has  provided  rigorous  and  explicit 
definitions  with  analytical  utility  in  mind,  and  intends  to  provide  specific  guidelines 
for  implementation  of  the  definitions. 

Tables  A-1  and  A-2  below  show  the  correspondence  between  the 
attributes  for  which  the  CRD  established  goals  and  the  attributes  that  will  be  used 
by  the  SIAP  SE  TF.  As  already  noted  in  the  Introduction  to  this  report,  these 
correspondences  refer  to  the  currently  valid  (2001  publication)  CRDs.  The 
correspondences  wilt  be  much  closer  with  the  next  revisions  of  these  documents. 

Table  A-1 

Correspondence  of  TAMD  CRD  KPPs  to  SIAP  Attributes 


TAMD  CRD  KPP 

SIAP  Attribute 

Completeness  (2  of  3  KPPs) 

Completeness 

Ambiguity:  (1)Tracks  per  Target, 

(2)  Fraction  of  Tracks  Representing  Truth  Objects 

Clarity 

(1)  Ambiguity  (Tracks 
per  Object) 

(2)  Spurious  Tracks 

Continuity 

Continuity 

(1)  Longest  Track 
Segment 

(2)  Rate  of  Track 

Number  Changes 

Position  Accuracy 

Kinematic  Accuracy 

Commonality 

Timeliness 

All  Attributes 
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Table  A-2 


Correspondence  of  CID  CRD  KPPs  to  SIAP  Attributes 


CID  CRD  KPP 

SIAP  Attribute 

Probability  of  ID 

ID  Completeness 

Correctness  of  ID 

ID  Correctness 

ID  Clarity 

Timeliness 

Ail  Attributes 

The  discussion  below  provides  a  more  detailed  description  of  the  attributes  and 
their  corresponding  CRD  KPP  where  applicable. 

Completeness 

The  TAMD  CRD  definition  of  completeness  is; 

Completeness: 

The  measure  of  the  portion  of  ground  truth  tracks  that  are  included 
in  the  SIAP.  The  metrics  that  describe  this  attribute  are:  percent  of 
threat  objects  in  track  upon  entering  the  area  of  influence;  percent 
of  total  objects  in  track  upon  entering  the  area  of  influence;  and 
percent  of  primary  and  secondary  systems,  as  defined  by  the 
scenario,  to  which  tracks  are  available  and  exploitable. 

The  TAMD  CRD  provides  three  KPPs  for  Completeness,  two  of  which 
relate  to  the  SIAP  attribute  of  completeness: 

XX  %  of  (T),  YY%  (O)  of  ground  truth  threat  objects  detected  and 
tracked  upon  entering  area  of  influence  (TBR),  and 
XX  %  of  (T),  yy%(0)  of  ground  truth  aircraft  objects  detected  and 
tracked  upon  entering  area  of  influence  (TBR). 

The  SIAP  SE  TF  has  provided  the  following  corresponding  definition  for 
completeness: 

Completeness:  The  air  picture  is  complete  when  all  objects  are  detected, 
tracked  and  reported. 

The  phrases  quoted  from  the  CRD  and  the  SIAP  SE  TF  definitions  include  the 
same  objects,  AOI,  and  implications,  and  are  essentially  identical  in  intent. 
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(There  is  a  third  KPP  metric  for  Completeness  in  the  2001  TAMD  CRD,  which  the 
2003  revision  proposes  to  remove.) 

(Clarity)  Ambiguity 

The  TAMD  CRD  definition  of  ambiguity  Is; 

Ambiguity: 

The  measure  of  the  clarity  of  tracks  in  the  SIAP.  The  metrics  that 
describe  this  attribute  are:  average  number  of  SIAP  tracks/air 
objects  at  any  given  time;  and  percent  of  network  tracks  that 
represent  distinct  ground  truth  objects. 

The  TAMD  CRD  provides  the  following  KPPs  for  Ambiguity; 

XX  (T),  YY  (O)  SIAP  tracks  per  air  object  (TBR) 

XX  %  (T),  YY%  (O)  of  tracks  represent  distinct  ground  truth  objects 
(TBR)  . 

The  SIAP  SE  TF  has  provided  the  following  corresponding  definition  for 

clarity: 

Clarity:  The  air  picture  is  clear  if  it  does  not  include  ambiguous  or  spurious 
tracks. 

Ambiguous  Tracks:  Tracks  are  ambiguous  when  more  than  one  track,  assigned 
to  the  same  object,  is  displayable  to  some  participant. 

Spurious  Tracks:  A  track  is  spurious  when  it  is  not  assigned  to  any  object. 

The  SIAP  SE  TF  definition  of  clarity  is  intended  to  be  more  explicit  than 
the  TAMD  CRD  definition;  furthermore,  the  issue  of  spurious  network  tracks  is 
separated  from  the  issue  of  ambiguity  (dual  tracks). 

Continuity 

The  SIAP  SE  TF  has  provided  the  following  definition  for  continuity: 

Continuity:  The  air  picture  is  continuous  when  the  track  number  assigned 
to  an  object  does  not  change. 
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Accuracy 


There  are  multiple  mission-dependent  definitions  of  accuracy  in  the  TAMD 
CRD  and  CID  CRD,  but  no  mission  specific  definition  for  air  objects.  An  example 
of  a  mission-dependent  definition  is  as  follows  for  BMD: 

Impact  point  accuracy; 

The  distance  from  actual  missile  impact  to  predicted  missile  impact 
point.  Methodologies  for  establishing  impact  point  accuracy  may 
take  a  number  forms:  An  ellipse  with  uncertainties  along  the  semi- 
major  and  semi-minor  axes,  an  azimuth  “wedge”  subtended  by 
minimum  and  maximum  predicted  range  limits,  a  geometric  figure 
that  contains  impact  point  predictions  and  associated  errors  from  all 
early  warning  elements.  (USSPACECOM) 

The  TAMD  CRD  provides  the  following  KPP  for  accuracy: 


Impact  Point  Accuracy  (XX  seconds  after  BO) 

<  YY  km  radius  circle  with  a 

ZZ  confidence  (T) 

Impact  Point  Accuracy:  (at  >  XXX  seconds  to 
impact) 

<  YYY  km  diameter  circle  (T), 
YYYY  km  radius  circle  (O) 
with  a  ZZ  confidence  (O) 

The  CID  CRD  also  refers  to  position  or  geo-location  accuracy  but  currently 
only  specifies  the  acceptable  allowance  for  increased  error  due  to  transmission 
of  CID  information. 

Geo-spatial  Accuracy:  “CID  systems  must  be  sufficiently  accurate  to  precisely 
correlate  characterizations  among  multiple  closely  spaced  surface  targets.  ” 

Geo-spatial  accuracy  will  be  met  if  all  participants  can  correctly  correlate  with 
network  tracks.  Thus  if  participants  are  not  generating  dual  designations  and  not 
miscorrelating  tracks  then  they  must  be  meeting  the  CID  requirement  for  geo¬ 
spatial  accuracy. 

The  SIAP  SE  TF  provides  for  a  combined  position  and  velocity  definition 
of  accuracy  as  follows. 

Kinematic  Accuracy.  The  air  picture  is  kinematically  accurate  when  the  position 
and  velocity  of  each  assigned  track  agree  with  the  position  and  velocity  of  the 
associated  object. 

The  desired  value  of  the  track  accuracy  is  mission-dependent. 
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Timeliness 


As  discussed  in  Section  2. 3. 1.5,  timeliness  is  not  included  as  one  of  the 
SIAP  attributes.  The  current  TAMD  CRD  does  not  specify  a  standard  for  the 
timeliness  of  track  data  and  the  Information  Exchange  Requirement  (lER)  for 
track  data  is  still  to  be  determined.  If  the  air  picture  is  complete,  clear,  and 
accurate,  then  timeliness  criteria  will  be  met. 

Recall  that  the  CID  CRD  specifies  a  standard  KPP  for  the  timeliness  of 
CID  data; 

The  CID  process  must  be  accomplished: 

a.  Soon  enough  to  provide  military  options  including  warning 
and  engagement  decisions. 

b.  Without  increasing  the  engagement  time  for  any  weapon 
system  more  than  XX  seconds. 

The  SIAP  SE  TF  has  not  assigned  a  SIAP  attribute  that  corresponds  to  this  KPP. 
However,  as  the  assessment  of  warfighter  benefits  proceeds  within  the  context  of 
specific  missions,  a  Measure  of  Effectiveness  (MCE)  will  be  associated  with  this 
KPP  to  assess  whether  warfighter  needs  are  being  met. 


CID  Attributes: 


The  CID  CRD  provides  the  following  description  of  combat  identification 
attributes; 

Combat  identification  is  the  process  of  attaining  an  accurate 
characterization  of  detected  objects  in  the  joint  battlespace  to  the 
extent  that  high  confidence,  timely  application  of  military  options 
and  weapon  resources  can  occur.  Depending  on  the  situation  and 
operational  decisions  that  must  be  made,  this  characterization  will 
at  least,  but  will  not  be  limited  to,  ‘friend,  ’  ‘enemy,  ’  or  ‘neutral’. 

Additional  characterizations  may  be  required  including  class,  type, 
nationality  and  mission  configuration. 

The  CID  CRD  KPPs  for  the  characterization  with  respect  to  ‘friend,’ 
‘enemy,’  or  ‘neutral’  are  given  below; 

(1 )  CID  Identification  probabilities  are  specified  for  Friends  (XX), 
Hostiies  (YY)  and  Neutrals  (ZZ). 

(2)  CID  probability  of  correct  identification  for  each  object  or  entity 
characterized  by  CID  Systems  QQ%  (T),  W%  (O). 
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The  SIAP  SE  TF  provides  three  ID  attributes  with  the  following  definitions: 

ID  Correctness:  The  ID  is  correct  when  all  tracked  objects  are  in  the  correct  ID 
state. 

ID  Completeness:  The  ID  is  complete  when  all  tracked  objects  are  in  an 
identified  state. 

ID  Clarity:  The  ID  is  dear  if  no  tracked  object  is  in  the  ambiguous  ID  state. 

The  CID  Identification  probabilities  specified  in  the  CRD  determine  the 
fractional  sizes  of  the  objects  that  remain  unknown.  Thus  this  is  a  form  of 
completeness  of  the  CID  picture,  broken  out  by  force  allegiance.  The  correct 
identification  probability  as  defined  in  the  CRD  is  a  ratio  of  the  SIAP  attribute 
measure  for  overall  CID  correctness  to  that  for  overall  CID  completeness;  i.e., 

IDc 

QQ  =  (threshold  requirement). 


The  following  rationale  of  the  KPP  values  is  given; 

Ideally,  the  CID  [systems]  should  be  capable  of  identifying  all 
detected  objects  and  entities  of  interest,  a  high  percentage  of 
“unknowns"  not  being  conducive  to  achieving  high  levels  of  military 
effectiveness  and  minimizing  fratricide  and  collateral  damage.  The 
different  thresholds  for  friend,  enemy,  and  neutral,  reflect  the 
following  assumptions:  a)  to  avoid  fratricide  and  to  expedite 
command  and  control,  it  is  important  to  identify  friends,  and  it 
should  be  technologically  easiest,  b)  To  identify  enemy  forces  is 
technologically  more  challenging;  therefore,  a  lower  identification 
probability  will  be  tolerated  in  order  to  achieve  the  desired 
probability  of  correct  identification,  c)  Since  the  process  of 
identifying  neutrals  may  be  the  greatest  challenge,  it  has  been 
assigned  the  lowest  threshold. 
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APPENDIX  B 


A  General  Mathematical  Procedure  for  Averaging  SIAP  Attribute  Measures 

This  appendix  outlines  a  general  procedure  for  carrying  out  various  levels 
of  averaging  and  obtaining  an  overall  "roll-up"  average  for  any  SIAP  attribute 
measure.  All  of  the  averaged  metrics  defined  in  Section  2.3  of  this  report  are 
derivable  as  special  instances  of  this  procedure. 

It  should  be  noted  that  the  expression  roH-up  metric,  also  used  in  SIAP  SE 
TF  Technical  Report  2002-007,  is  not  always  understood  to  connote  an  average, 
and  for  certain  ballistic  missile  SIAP  metrics  the  term  is  used  in  a  different  sense. 
However,  for  all  of  the  air  vehicle  SIAP  attribute  measures  discussed  in  this 
report,  the  roll-up  always  refers  to  the  highest-level  average  over  participants, 
time,  and  objects/tracks  as  appropriate. 


To  accommodate  evaluation  of  the  metrics  in  simulation  environments 
where  multiple  runs  may  be  executed  (Monte  Carlo  simulations),  averages  over 
runs  are  also  embedded  in  the  roll-up  procedures. 

It  is  expected  that  the  user  will  define  the  time  interval  over  which  the 
SIAP  attribute  measures  are  to  be  evaluated  (and  averaged,  if  appropriate),  as 
well  as  a  schedule  of  scoring  times.  When  the  purpose  of  the  evaluation  is  to 
assess  compliance  with  requirements,  the  interval  of  evaluation,  the  division  into 
scoring  subintervals,  and  the  scheme  for  selecting  scoring  times  should  be 
constant  for  the  scoring  of  all  SIAP  attribute  measures  for  a  given  CRS  and  in 
accordance  with  the  analysis  plan.  However,  for  other  diagnostic  purposes,  a 
scoring  schedule  tailored  to  particular  events  or  objects  of  interest,  or  possibly 
varying  between  different  metrics,  may  be  appropriate.  The  averaging  formalism 
is  therefore  set  up  so  as  to  be  indifferent  to  the  time  period  of  evaluation,  which 
may  be  specified  by  the  user  to  be  object-  or  participant-  or  metric-specific. 

The  general  procedure  for  rolling  up  a  SIAP  attribute  measure  may  be 
described  as  follows.  Suppose  a  single  weighted  average  value  is  required  for  a 
metric  Vj,m,n(tk),  the  metric  being  defined  in  a  way  that  is  specific  to  the  object  (j), 
participant  (m),  scoring  time  (tk)  and  run  (n)  being  assessed.  For  some  values  of 
the  arguments,  the  metric  may  possibly  be  undefined  (perhaps,  for  example,  if  j 
designates  an  object  on  which  participant  m  holds  no  assigned  track).  Let  the 
values  designated  by  Wj.m.n.k  be  a  set  of  nonnegative  weights  which  are 
meaningful  and  appropriate  for  all  values  of  0,m,n,k)  for  which  Vj,m,n(tk)  is  defined. 
In  the  formulae  for  specific  attribute  measures  which  follow  in  the  remainder  of 
this  Appendix,  the  weighting  may  be  by  number  of  objects  or  events,  number  of 
participating  units,  or  reportable  object  lifetimes,  as  applicable.  By  convention, 
set  Wj,m,n,k  equal  to  0  if  either  (1 )  Vj,m,n(tk)  is  not  defined,  or  (2)  the  scoring  time  tk 
is  not  contained  in  the  time  interval(s)  of  interest  for  this  metric  as  considered  for 
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object  j,  participant  m,  and  run  n.  The  following  roll-up  average  over  runs, 
objects,  participants,  and  time  then  simply  excludes  all  measurements  which  are 
either  not  of  interest  or  not  defined: 


K  M  J  N 


V 


t<=t  m=1  j=1  n=1 


K  M  J  N 


(B-1) 


ksl  m=1  ji-1  n=1 


where  K  is  the  total  number  of  scoring  time  intervals,  M  the  number  of 
participants,  J  the  number  of  objects  occurring  in  the  scenario,  and  N  the  number 
of  runs.  This  formulation  has  the  additional  mathematical  advantage  that  the  four 
individual  averages  that  it  encompasses  can  be  carried  out  in  any  order.  This 
feature  allows  for  considerable  flexibility  in  the  definitions  of  intermediate 
averages  on  the  way  to  the  final  roll-up.  For  example,  if  a  system  time-averaged 
value  Vm  specific  to  participant  m  is  required,  one  simply  omits  the  summations 
over  m  in  the  above  formula.  The  final  roll-up  value  V  can  then  be  re-formulated 
as  a  weighted  average  of  the  individual  V^s  over  participants,  with  the  weights 

K  J  N 

being  ■  Alternatively,  if  an  instantaneous  IADS  average  V(tk) 

k=1  j=1  n-1 

over  participants  is  required,  this  can  be  calculated  from  the  formula  obtained  by 
omitting  the  sums  over  k  in  the  general  roll-up  formula.  The  roll-up  value  V  is 
interpreted  as  the  weighted  average  of  the  individual  V(tk)s  over  scoring  times, 

M  J  N 

with  the  weights  now  given  as  ■  The  flexibility  as  to  intermediate 

m=:1  j=1  n=i 

averages  may  prove  useful,  not  only  for  various  supplementary  analysis 
purposes,  but  also  in  adapting  the  attribute  measures  to  support  differing 
interpretations  of  CRD  requirements  (until  a  common  understanding  is  reached) 
or  changes  to  the  requirements  themselves  in  future  CRD  updates. 

The  identity  of  the  weight  factor  Wj,m,n,k  for  a  given  attribute  measure  may 
be  ascertained  from  the  formula  defining  the  fundamental  instantaneous  metric 
for  the  attribute  (in  the  case  of  continuity,  the  object-specific  metric)  in  the  body  of 
this  report.  As  a  specific  example,  for  the  instantaneous  system  measure  of 
Ambiguous  Tracks  given  in  equation  4  (one  of  the  clarity  attribute  measures), 
assuming  for  simplicity  that  the  measure  is  evaluated  for  a  single  run  (n=1),  the 
weight  factor  Wj.m,i.k  is  the  number  of  objects  tracked  by  participant  m  at  the  k*^ 
scoring  time  ~  JTm(tk)  -  and  is  independent  of  the  index  j.  That  is,  the  weight  is 
given  by  the  denominator  in  the  expression  for  the  instantaneous  system  metric 
Am(tk)  in  equation  4.  The  application  of  the  general  procedure  to  this  case  is  as 
follows.  The  sums  over  runs  (n)  in  the  equation  B-1  do  not  apply  since  n  has 
only  one  value,  and  the  sums  over  objects  (j)  have  no  effect  since  both  the 
weight  factor  and  the  instantaneous  system  metric  are  defined  independently  of  j. 
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Applying  the  sums  over  time  steps  (k)  alone  produces  the  time-averaged  system 
metric  as  in  equation  5: 


K  K 

^  jTm(t,^  )A^  (tk  )At,,.  ^  NAm(tk  )Atk 

A.  =  -  .  (B-2) 

£jTr.(tk)Atk  |;jT,.(tk)Atk 

k=1  k=1 

while  separately  applying  the  sums  over  participants  (m)  alone  produces  the 
instantaneous  IADS  metric! 


M  M 

^jTm(tk)Am(tk)  ^NAm(tk) 

=  - •  (B-3) 

ZjTniCtk) 

m=1  fin=1 


The  roll-up  metric  A  of  equation  6  (a  specialization  of  equation  B-1)  may  then  be 
interpreted  as  a  weighted  average  of  Am  over  participants,  or  a  weighted  average 
of  A(tk)  over  time  steps.  The  invariance  of  the  roll-up  metric  with  respect  to  the 
order  of  averaging  is  illustrated  by  the  numerical  example  displayed  in  Table  B-1 . 
The  term  “track”  in  this  table  is  understood  to  mean  an  assigned  track,  and 
“object”  is  understood  to  mean  a  tracked  object  (that  is,  a  reportable  object  to 
which  at  least  one  track  is  assigned).  The  notation  x/y  in  a  table  entry  means  x 
tracks  and  y  objects  (that  is,  x  assigned  tracks  and  y  tracked  reportable  objects). 
A  0/0  entry  signifies  any  of  the  following  cases:  that  there  were  no  tracked 
objects  (and  thus  no  assigned  tracks),  that  no  data  was  available  for  the  given 
participant  at  the  given  time,  or  that  the  participant  was  attritted  by  the  given  time. 
Every  averaging  step  consists  of  independently  summing  the  track  and  object 
counts  in  the  table  entries,  either  across  a  row  or  down  a  column  of  the  table. 
Scoring  is  assumed  to  be  at  uniform  time  intervals. 
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Table  B-1 :  Sample  Averaging  Calculations  for 
Ambiguous  Tracks 


34/27  «  1.26 


(derivable  from  either 
the  system  averages  or 
instantaneous  averages, 
through  the  same 
averaging  procedure) 


All  of  the  SIAP  attribute  measures  discussed  in  the  body  of  this  report, 
with  the  exception  of  the  kinematic  accuracy  measures,  follow  exactly  the  same 
procedure  with  respect  to  averaging.  The  slight  exception  for  kinematic  accuracy 
arises  from  the  use  of  RMS  averages  for  position  and  velocity  errors,  in 
accordance  with  standard  practice  in  error  statistics.  However,  if  the  squares  of 
the  position  and  velocity  errors  are  regarded  as  the  fundamental  metrics  of 
interest,  then  the  squares  of  the  kinematic  accuracy  attribute  measures  defined 
by  equations  15-22  follow  exactly  the  pattern  laid  out  above. 
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